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ABSTRACT 


The  peet  15  years  have  seen  Important  advances  in  the  understanding  of 
behavioral  decision  processes.  Despite  these  advances  there  has  been 
surprisingly  little  attention  paid  to  the  training  of  decision  makers. 

This  report  provides  a  summary  of  a  conference  held  In  July  1976  In 
Eugene,  Oregon  to  evaluate  the  state  of  knowledge  regarding  the  train¬ 
ing  of  decision  makers.  Identity  Important  research  questions  and  recommend 
directions  for  future  research.  The  report  provides  both  a  brief  review 
of  current  training  programs.  Including  a  discussion  of  kev  attitudes 
and  ekllls  required  for  decision  making,  and  a  list  of  the  most  important 
research  questions  that  emerged  during  the  conference;  Three  aspects  of 
the  training  problem  recommended  for  Immediate  attention  were  (1)  training 
apeclflc  skills,  (2)  evaluating  the  quality  of  decisions,  and  (3)  implementing 
the  knowledge  obtained  through  decision  research. 
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I.  Introduction  and  General  Issues 


The  rise  of  automation  and  the  increased  pace  and  intensity  of  con¬ 
temporary  conflict  have  made  intellectual  skills,  especially  those  of 
judgment  and  decision  making,  crucial  elements  in  the  functioning  of 
military  and  defense  systems.  Because  of  this,  a  considerable  amount  of 
recent  research  has  attempted  to  describe  the  cognitive  processes  of 
decision  makers  and  to  develop  technigues  for  improving  decisions. 

Despite  this  effort,  there  has  been  surprisingly  little  attention  paid 
to  the  training  of  decision  makers.  The  present  conference  was  convened 
to  tackle  this  problem  directly,  with  an  eye  towards  recommending  direc¬ 
tions  for  future  research. 

One  guestion  that  this  conference  addressed  was  "What  do  we  know 
about  training  decision  makers?"  The  question  proved  more  complex  than 
it  first  appeared  to  be.  A  qreat  deal  of  time  was  spent  In  defining 
necessary  terms,  establishing  appropriate  boi  faries,  and  in  essence, 
redefininq  the  question. 

Who  i s  a  Dec i s i on  Maker ? 

It  was  first  necessary  to  define  the  locus  of  decision  making.  Is 
the  decision  maker  a  single  individual  or  a  group?  Which  person  in  a 
qroup  or  organization  should  be  trained?  In  resolving  these  issues,  we 
aqreed  on  the  following  points:  (I)  decision  making  is  often  an  inter¬ 
active  process  within  an  organ i zat ion ;  the  training  of  a  lone  decision 
maker  will  be  different  from  the  training  of  a  decision  maker  and  his 
or  her  supporting  staff;  (2)  when  complex  situations  force  decision 
makers  to  assign  subparts  of  the  decision  task  to  others,  the  added 
common i ca t i on  and  monitoring  must  be  considered  in  the  training  program; 
(3)  for  the  purposes  of  this  conference,  a  decision  maker  is  the  lowest 
level  person  within  an  organization  who  could  "sign  a  check"  for/or 
authorize  the  distribution  of  resources.  In  this  perspective  we  were 
concerned  with  tactical  as  opposed  to  strategic  or  operational  decisions. 
Decision  makers  falling  within  our  definition  include  the  squad  leader 
or  military  commander  mak i nq  tactical  decisions  on  the  basis  of  rapidly 
received,  incomplete  and  unreliable,  and  sometimes  irrelevant  information. 


Genera  I  Education  vs  . _ Sj><  <  ij  ij  T  r  n  i  i»g 

Should  decision  makers  be  educated  in  general  principles,  or  trained 
in  techniques  applicable  to  t h<- .  i  special  situations,  or  both?  This 
empirical  question  will  only  -olved  by  experiments  i  nves  t  i  qa  t  i  nq  the 
relative  effectiveness,  retention  and  t i ansferabi I i ty  of  training  regimens 
with  different  balance.  if  general  iciples  and  specific  techniques. 

Five  levels  '  f  Ira  .lag,  «  •  a  with  its  respective  goals  were  differ¬ 
ent  iated  : 

(1)  The  altitudinal  li-.el  training  designed  to  qet  the  decision 
maker  to  believe  that  better  dei  isi.  ns  arise  fiom  decomposing  a  problem 
into  it.  relevant  parts  .<  •  d  ,v,se  :  the  relevant  uncertainties  and 
utilities. 

(2)  The  aids  level  the  dec i  is  naker  accepts  the  decision  theo¬ 
retic  philosophy  and  learns  to  pro-,  ide  the  necessary  inputs  into  pre¬ 
constructed  models. 

I J)  The  cookbook  level  te.  .1  the  de  i s i on  maker  to  follow 
simple,  explicit  rule  *  ■  ;  V.  is  .ms.  To  avoid  trie  danger  of 

"a  little  knowledge  in  t  ?er  ■  thin,,"  training  for  this  level 
of  achievement  would  have  !  11  lute  explicit  warnings  of  the  kind  "If 

your  problem  has  any  >*f  thee  elements,  <  ill  your  well-trained  decision 
analyst 

(I*)  The  for  i.il  a«i  >  t1  iy  level  teaching  the  decision  maker 

not  on  I  y  how  to  do  a  feci'  '■  ana  ly  si.,  hut  !  ‘  <■  f-rri.il  axiomatic  theory 

behind  it  as  we  I  I . 

(5)  The  expert  level  the  level  of  t t a i n i nq  needed  to  practice  the 
art  to  the  limit  of  its  present  state  of  development. 

Task  Parameter  s 

Feedback.  Before  dec 1  ling  on  the  curriculum  for  training  a  specific 
decision  maker,  several  ispci  tx  of  his  or  her  task  must  be  taken  into 
account.  One  aspect  is  the  kind  of  feedback  available.  Two  important 
feedback  dimensions  are  validity  and  ti  e  lag,  which  taken  together, 
determine  feedback  effectiveness. 

Performance  c  r  i  t  e  r  i  a  .  ■'>  crucial  aspect  of  any  training  program  is 

the  ability  to  evaluate  it'  success  For  prediction  tasks  where  people 
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dk<*  *  uecasts  about  future  eve  t-.  hu  past  events  whose  outcomes  are 

le  correct 

I  'My 

t.  s,s  .  it'  tv  i  li  t.  >  ) ;  ;  i  1 1  1  i  <'■  iuq  i  i  •  lat  I  vc '  y  easy  wi  t  h  a  stral  qht- 

fgrwai  efore . ■  i •  1 1  lesign.  I f  subjects  score  sign I f I cent tv 

bettei  .if  let  {ruining,  i  we  an  . one  I ude  tti.il  they  have  learned  some- 

:  ibout  this  t  ■■  Mhechet  "something"  is  also  "enough" 

is  a  separate  question.  To  determine  whether  the  traininq  conferred  a 
true  .►ill,  •  I  .  :  t  ainin.)  easures  on  different  tasks 

in  different  contexts  are  needed. 

The  real  ev  it  *  •  in,;  n  situations  where  there 

j  ,  nit  I*.  One  e  •  hod  is  to  construct 

luati  :<■ 

... 

t  r>  the  r.-  a  I  .vc  -  I  •  “  ■  t  i  a  I  ques  t  i  ■  with  this  approach  i  s  "Have 

you  built  suffi  ic<  '  scihrl  :  and  t.v  |>le*i.y  into  the  laboratory  task 
such  that  "i  haw  faith  i  t  >e  ,enc' al I /abl 1  i  ty  of  the  results?" 

A  .  .i  f  I'.-;.  «•  *  pe :  t  jttdqes  o  peer  ratings  to  determine 

t  wa  I  i  t  y  T  ,m  .  a  r  <•  •  •  > .  i  i  .<•.  ,.f  .teer-rr.  its.  approaches,  the  process- 

»r .  •  f ,  t-<  . ;  > 1  i .  it  t  he  nice  ■••of  ic-nted.  With  the  process- 

,  tab  I  edui  tch  t  ha  t 

<ie  i  s  ikeis  fr  I  l<  .  •  he  e  rale,  are  considered  ;ood  decisionmakers. 

The  proble  or  ir  ted  tppt  i.ich  tries  t  identify  and  correct  recurrent 
proSle'-s  The  out  •  i.-ntrd  approach  i  mcent  rates  on  ev.  •’  .at  inq  the 

Outcomes  of  the  dr  is  i  t>w  laker's  previous  lecislons.  There  are  problems 
with  each  of  these  approaches. 

The  major  flaw  ir  : ne  ,  nccss*  and.  the  problem-oriented  approaches 
ts  tbit  the  ev  i  e  e  of  iprr.  ed  de  i  ion  -••king  is  indirect.  One  cannot 

west  , 

followinq  a  proaram,  in  order  to  determine  its  cost-ef feet i veness .  The 

a  beb 1 1  istlc 

niter  r.  ■  ■ .)  nu  t  r  o  an  result  ‘  rr>m  bad  fer  i  s  i  ons  and  bad  On  t  comes 

Thu  .  ntess  a  large  sample  >*  i  c'-i  t  s  tvai labia, 

one  can't  always  judge  the  quality  of  choices  solely  from  the  results 
produc.rd . 


As  .1  result  of  the  above  considerations,  the  conference  question, 
"Wh.it  do  we  know  about  training  decision  makers?"  was  redefined  as  "What 
is  the  present  state  of  knowledge  about  training  decision  makers  and 
their  staff  to  make  'ore  effective  and/or  efficient  tactical  decisions 
when  tne  decision  makers  may  not  have  access  to  the  outcomes  of  previous 
dec i s ions?" 


II.  The  Current  State  of  Knowledge 
Current  f raining  Programs 

Although  the  training  programs  described  here  all  have  the  general 
aim  ot  aiding  of  decison  nkers,  they  differ  in  length  of  the  training 
offered,  specific  ■  uterial  covered,  and  depth  of  coverage.  None  has 
been  tested  relative  O'  absolute  effectiveness,  nor  for  the  retention 

or  the  transferability  of  its  training. 

An  Id!  le  to  Decision  Making.  An  approach  to  teaching  multi* 

attribute  tile'/  analysis  t  ■  top-level  executives  and  middle-level  managers 
has  been  developed  by  Hi  I  lei  E inhorn  and  Robin  Hogarth.  The  Guide  provides 
a  set  of  simple  rules  with  which  the  users  can  perform  their  own  multi- 
attribute  utility  analyses  and  a  set  of  warning  signals  which  indicate 
when  the  r  lies  should  not  be  used  because  the  model  underlying  them  is 
violated.  The  Guide  is  taught  in  an  hour  and  a  half  seminar;  no  formal 
theory  or  rationale  for  the  rules  is  riven.  The  program  is  sold  by  means 
of  mu  I t i a 1 1 r i but c  examples  selected  to  shake  the  decision  maker's  confidence 
in  his  or  her  clinical  judg-ient  and  to  produce  a  perceived  need  fot 
training.  The  examples  are  carefully  selected  for  relevance  and  interest. 
For  example,  one  concerns  the  utility  of  individual  professional  basket- 
bal I  players. 

The  ' Idiot's  Guide"  recommends  the  following  steps: 

1.  List  all  the  important  attributes  of  the  options  you  are  consid¬ 
ering.  Try  not  to  add  attributes  highly  correlated  with  others  already 
chosen . 

2.  Define  all  the  attributes  so  that  they  are  scaled  nonoton i ca I  I y 
in  a  positive  direction  with  respect  to  total  worth,  making  certain  that 
all  attributes  are  conditionally  monotonic. 


!> 


3.  Weiqht  .1 1 I  attributes  equally  (i(  you  are  very  unhappy  with  this, 
use  a  weighting  syslt-  based  on  a  rank  order  of  importance). 

Fori  i  matrix  with  the  alternatives  in  the  rows  and  the  attributes 
in  the  columns.  Fill  the  ••  I  I  entries  with  ratings  (which  in  some  cases 
might  be  hard  numbers  like  dollars). 

5.  Calculate  '-an  a  J  ti’dard  deviation  ot  each  attribute 

(column) . 

6.  Stai  lardi/i  each  cell  entry  (subtract  the  column  mean  and  divide 
by  the  standard  deviation). 

/.  Multiply  e.  h  standardised  cell  entry  by  the  weight  assigned  to 
the  a  1 1  r i bote . 

8.  $u  th>  we  ;hte  !  standardised  scores  for  each  alternative. 

9.  Sell  t  eithc  the  »lte’ native  with  the  highest  score  or  that 
alternative  with  the  hi  jhrst  benefit  to  cost  ratio. 

Si  pi  ■  f  ic  at  '  i  '  lei  i  "d  task'  r(o  rules  such  as  those  in  the 

it  ionale .  For  «•  unp l  e , 

Linhorn  >*  •  :  h  have  shown  ’.he  effectiveness  of  equal  weights 

in  additive  ->o<Te  I  s . 

hniques  t ha t  0e<  i  >  t on  and  Oes i gns ,  Inc. 

■  ODI  )  ises  f tii  s  ,,  i  d<-i  i  .ion  throretic  concepts  is  practice 

in  ondu.,  tin:;  s  i  ,  I  ••  ‘  lr  >  .  analyses.  The  approach  involves  an 

interactive  o'cute  jr.i  that  simplifies  a  decision  analysis  by  focus- 

inn  on  a  li  ited  ■■■>  •  of  liter  nji  the  ma ior  causes  of  uncertainty 

S  that 

the  probabilities  are  independent  of  actions.  Decision  makers  start  out 
by  identifying  the  fow  1 1  ter nat i ves  and  the  one  particular  uncertain  event 
that  they  will  he  corsi  irrir  .  Trainees  construct  a  value  matrix  similar 
to  step  four  of  tinhorn's  procedure,  except  that  cell  entries  are  reqrets. 

the  dec  Is  ion  maker  decide  first  on  the  best 
option.  All  othei  ,  :  >ns  ire  rated  in  comparison  to  this  option.  Once 
this  ha.  been  don.-  f  ir  every  attribute,  the  attributes  are  weiqhted,  not 
necessarily  equally,  v.  t  then  combined  addi lively.  The  computer  then 
performs  the  necessary  calculations  and  displays  the  expected  value  of 
each  alternative  Sensitivity  analyses  are  also  demonstrated  to  show  the 
implications  of  changed  assessments. 
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Spec i a I  Judgment  Training.  Hammond's  (1971)  approach,  via  social 
judgment  theory,  assumes  that  if  people  are  taught  the  theory  behind 
judgment  analysis  and  then  trained  in  increasingly  difficult  applications 
of  task  situations,  they  will  eventually  be  able  to  analyze  any  problem 
properly.  In  practice  sessions,  trainees  work  with  a  computer  terminal 
display  initially  presenting  a  simple  task,  involving,  say,  two  cues. 

They  arc  asked  to  use  a  linear  rule  to  combine  these  cues.  As  training, 
with  model  feedback,  proceeds,  the  tasks  are  complicated  by  using  different 
weighting  schemes,  more  variables,  and  different  functional  relationships. 
After  several  days  of  such  training,  people  can  do  well  on  very  complicated 
tasks.  In  the  training  up  to  this  point,  the  variables  are  all  specified. 

In  the  next  step  of  training,  people  learn  to  uncover  the  structure  of 
variables  in  different  kinds  of  tasks.  Here,  the  tasks  are  presented  in 
written  for  and  t he  trainees  must  extract  the  variables  and  determine 
the  structure  of  these  increasingly  less  we  I  I -spec i f ied  problems.  The 
final  pha  ,e  of  training  consists  of  having  people  evaluate  the  task 
structure  of  general  statements  found  in  print  such  as  a  policy  statement 
written  by  Kissinger.  Related  procedures  enable  opposing  sides  in  a 
conflict  situation  to  carefully  specify  their  policies  and  identify  the 
exact  Sources  of  d i sag rcement .  In  several  field  studies,  this  approach 
derived  from  social  judgment  theory  (Hanrtond  et.  al.,  1975.  1976)  has 
successfully  reduced  conflict  by  showing  it  to  be  less  than  imagined  orig- 
i na I  I y  (Hammond,  19  71;  Hammond  C  Boyle,  1970;  Hammond  t  Brehmer,  1973). 

Training  in  Decision  Analysis.  The  Decision  Analysis  Group  at  Stan¬ 
ford  Research  Institute  (SRI)  conducts  several  different  kinds  of  training 
programs  for  executives  and  staff.  There  are  one-day,  three-day,  and  five- 
day  public  seminars  or  private  courses.  The  private  courses  differ  from 
the  open  seminars  in  that  they  can  be  tailored  to  specific  organi zat ional 
needs.  The  one-day  program  attempts  to  convey  an  understanding  of  what 
decision  analysis  is  and  how  it  can  be  applied.  Topics  covered  include: 
introductory  decision  analysis,  problem  structuring,  probability  encoding, 
profit  lotteries  and  decision  trees,  information  valuation,  sensitivity 
analysis,  and  r i sk -pre f erence  encoding.  Applications  and  examples  are  a 
primary  vehicle  for  describing  the  concepts.  SRI  attempts  to  motivate  partic 
pants  by  having  them  work  through  examples  in  which  the  majority  of  unaided 
derision  makers  make  bad  judgments.  SRI  has  found  that  many  decision  makers 
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C(,iIlr.  {,,  these  seminars  because  they're  woriied  mut  uncertainty  and  don't 
know  how  to  handle  it.  Therefore,  they  are  usually  receptive  to  attempts 
to  help  them  think  loqically  about  uncertainty. 

In  the  three-  and  five-day  seminars  more  time  is  spent  on  all 
topics  and  the  decision  makers  actively  work  through  different  problems. 
About  half  a  day  is  spent  talking  about  biases  in  uncertainty  judgments 
and  in  teaching  participants  how  to  overcome  them.  Interactive  computer 
programs  allowim  the  exploration  of  decision  problems  without  the  need 
foi  complex  nanua  I  co  pu  tat  ions  are  demonstrated,  in  these  seminars, 
people  are  expected  t  learn  that  the  decision  theoretic  methodology 
exists,  that  urui"  t.rit  irs  and  utilities  cun  be  quantitatively  estimated, 
and  that  they  .an  (><■  ;  i  r.  to  structure  and  work  through  their  own  decision 
.•roblems  Three  lay  e-inars  are  ittended  by  hiqh-level  decision  makers. 
Since  middle- level  mi;."  and  staff  people  arc  responsible  for  actually 
••e-fo’  ng  analyses,  they  typ.  illy  attend  the  five-day  seminars  which 
.  in  v  •  de  t  fie"  .,  h  t  fie  t  .s'  tools  needed  to  benin  the  appren  t  i  ceshi  p 
state  of  learning  tiy  doing. 

HSU  »  Physician  Tr.i  n i nq  Program.  Michigan  State  University's  (MSU) 
Medical  School  appr  .a  h  to  the  training  of  physicians  as  described  by 
lee  Shul  an  is  base.'  iv  a  total  curriculum  design  wherein  decision  making 
tool--  are  nteoite'  into  the  study  of  specific  content  areas.  The  approach 
asvi  es  t*  it  the  d  vinos' ic  phase  of  sedicine  consists  of  generating 
hypothese  ahuut  w'at  the  x’dical  proble  sight  be,  distinguishing  the 
relevant  fum  •  >ie  i  rrv.tlrnt  features  of  the  as  c,  and  then  sy  s  t  ema  t  i  ca  I  I  y 
gather  in;  it  m  t<  test  and  ogi-iri  alternative  hypotheses.  The 

training  curriculum  based  on  this  conception  relies  heavily  on  the  casc- 
studv  method  Student  are  presented  with  cases  from  each  field  of  medicine 
seleited  either  for  frequency  of  occurrence  or  severity  of  consequences. 
Students  are  taught  i  .stem  of  medical  documentation,  called  problem- 
oriented  nrdical  recording.  This  method  preserves  the  process  by  which 
the  proble  was  wotked  up  and  analyzed,  and  becomes  itself  a  teaching  aid. 
Students  audit  eai».  others'  records,  thereby  gaining  additional  experience 
by  making  their  **vn  de.  isions  and  critiquinq  those  made  by  Others.  Thus, 
openness  to  analysis  and  criticism  by  one's  peers  is  taught  along  with 
sufirct  alter  and  > n f . > -mat i on  qatherinq  skills.  The  specific  form  of 
the  medical  record  ■  designed  as  a  decision  aid  to  structure  thinking. 


The  physician  is  prov  i  ded  w i t  h  h ■  d  .r  <  r-nirds  ot  hi  or 

her  e * pe  r  i  enc e  tor  t  utm  •  i  <•  t 

Two  questions  concerning  training  re  .•  t  relative  to  MSU’s 

program.  First,  shoo  I  <1  de<  ,  i<  •  -  .r . .  q  <  or  xt  rr  setting  specif  i 

Second,  would  formal  I  i  Jinirii  >  '  .  <■  t  »•  iseful,  except 

in  altering  decision  siker  to  ■  >  -r  vhi  .  among  the 

typical  phases  of  dr.  -.ion  v  '  *  •  ■  *  dr  i  •,  i  on  algo¬ 

rithms  can  be  most  use*  jl? 

Traininq  livestock  Ju.l  '  »  *  •  -t  at  Kansas 

State  University  runs  a  four  y«  >r  t  r  nil  j  ;>i  I  tock  judges. 

As  reported  by  Jane  s  ■  n  m  • ,  . ,  ,  ,  '  a  f  1 1  •  rd  i  •  ar  I  I  y 

•fftetlva.  For  exanp 

larger  number  of  judg  «•  it  i  v.o  abl-  u  ■  r  -d  elsewhere 

(at  least  10  to  II  dimen  ength 

-  -  four  year  s  .  soc  i  a  1  ,  t  rue  ’  ir  c  '  1 1  <  :<■  t  e  with 

roaches  and  rev  >pe  t  i  t  i ons ,  feedba  ►  •  <  I  r  r  .  their  judge¬ 
ments  and  the  reasons  leal  t  then  •  xpose 

their  cons  i  ife  ra  t  ion-  f  >  , .  >  *  r  >  .  '<  >n  -experts' 

judqnents .  It  Is  not  clear  which  <•'  t  •  •  i.  -  i  e  -I  to  their 

apparent  prowess  or  how  t>  "life  •  eri  ent"  >•  •  -r« ili/cd  for 

other  training  pro  r  > 

Training  I  he  t 

expert  analysts  can  he  t-  i  i  nrd  >'  ‘  <1,  ''’lastlc 

st  udy  and  prac  t  i  ca  I  r  ■ .  r  ,  ,-  v  r  t  >■  i  t  i  t  i  oner 

It  tMf  not  dtir  Mlwtl  1  i 

because  there  are  skills  involved  that  ■  ann  t  t  :n  the  classroom, 

or  because  t  be  forefront  of  tee  i  s  m  a  I ->  is  '  tro  too  ast  for 

testbooks  and  training  programs  ro  ray  irrent 

Training  Key  A  t  t  <  t  ude  s  arid  Skit' 

Generating  a  list  of  attitudes  ar t  .sills  needed  '  >r  decision  making, 
for  which  the  traininq  research  literature  mioht  lie  evirwed,  turned  out 
to  he  a  rather  difficult  task  The  followin  I  t  o*  attitudes  and  skills 
smnari/es  the  results  if  our  effort 
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1.  Recognizing  that  a  decision  problem  exists  and  that  it  needs 
to  be  solved. 

2.  Legitimizing  the  problem;  that  is,  getting  others  to  recognize 
that  the  problem  exists  and  requires  resolution. 

3.  Defining  the  level,  scope,  and  boundaries  of  a  specific  problem, 
k .  Identifying  t  he  problem  variables,  capturing  all  essentials, 

and  eliminating  redundanc i es . 

5.  Generating  alternatives. 

6.  Defining  criteria,  goals  or  value  dimensions  that  should  be 
included  in  the  f i na I  structure. 

7.  Selecting  the  structural  model  by  which  individual  judgments 
will  be  combined  into  a  composite  judgment. 

8.  Acquiring  information. 

9-  Dealing  with  uncertainty.  This  includes  accepting  uncertainty, 
encoding  probabi I i t i es  and  probability  distributions,  ar.d  avoid¬ 
ing  known  biases  (such  as  iqnorlng  base  rates). 

10.  Measuring  utility. 

11.  Combining  single  judgments  into  composite  judgments. 

12.  learning  from  experience;  that  is,  establishing  organ i za t iona I 
schemes  for  monitoring,  evaluating,  and  improving  one's  decisions. 

13.  Knowinq  when  and  how  to  use  heuristics,  short  Cuts  and  decision 

aids. 

Ik.  Knowing  one's  own  limitations  and  when  to  use  normative  experts. 

15-  Keepinq  focused  on  the  problem  to  be  solved. 

16.  Ccvtxnunicat  ing. 

1 7.  Implement ing. 

The  training  of  many  of  these  skills  has  yet  to  be  studied.  We  will 
describe  the  state  of  knowledge  on  training  for  each  category  separately, 
h,i\»*d  on  research  results  and  on  our  own  experience. 

Recognizing  a  Decision  Problem.  Disagreement  exists  about  whether 
recognizing  problems  is  a  general  or  contex t - spec i f i c  skill.  Some  research 
in  the  use  of  qroup  process  techniques  (e.g.,  nominal  groups,  Delphi,  and 
brai nstorming)  shows  that  people  can  be  trained  to  be  better  problem  Ident¬ 
ifiers.  However,  this  research  is  confined  to  situations  where  subjects' 
sole  task  is  isolating  problems  (Utterbach,  1 9 7^ ;  van  de  Ven  t  Delbecq,  1971). 
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We  have  heard  that  there  is  research  that  uses  Sherlock  Holmes  stories  to 
train  people  for  problem  sensitivity.  For  example,  one  point  illustrated 
by  holmes  is  to  be  as  alert  to  things  that  don't  happen  as  to  things  that 
do.  Another  research  area  is  the  Bayesian  research  on  people's  perception 
of  nonsta t i onar i t i es .  Chinnis  and  Peterson  (1968)  showed  that  people  can 
detect  change,  but  their  subjects  wete  specifically  made  Jwaie  ot  the 
possibility  of  change.  Consequently,  their  research  tells  us  nothinq  about 
people's  skill  at  detecting  unant i c i pa t ed  problems.  One  suggested  way 
to  discover  problems  is  to  be  alert  for  inconsistencies  or  disagreements 
among  experts.  For  example,  when  the  National  Board  of  Medical  Examiners 
found  that  different  experts'  evaluations  of  a  physician's  performance 
correlated  only  about  .20,  they  realized  that  their  evaluation  procedure 
had  to  be  changed.  Thus,  training  people  to  keep  records  of  judgments 
and  rev  ew  them  for  discrepancies  might  increase  their  ability  to  recognize 
a  decision  problem. 

Legitimizing  t  Ik~  problem.  We  had  little  to  say  about  this  topic. 
Several  references  on  the  importance  of  the  legitimacy  of  problem  solving 
in  an  o rgan i zat i ona I  context  are  Dclbecq  (1975),  Hage  and  Aiken  (1970), 
and  Shukla  (1976). 

Defining  the  Problem.  The  consensus  of  the  group  was  that  these  skills 
only  cone  with  experience-  Experimenters  at  Michigan  State  University 
(Allal,  1973;  Elstein,  Shulman  6  Sprafka,  1977)  conducted  a  study  showing 
that  the  speed  of  acquiring  this  skill  may  be  enhanced  by  having  the 
decision  maker  practice  defining  problems  previously  defined  by  experts. 
Their  trainees  examined  both  the  expert's  final  structuring  of  the  pit- 
and  transcribed  thought  processes.  No  test  of  neneral i zabi I i ty  ot  trans¬ 
ferability  was  incl  Jed  in  the  experiment. 

Generating  Problem  Variables.  In  we  1  I  -  8 1  rue  lured  situation-.,  people 
apparently  can  learn  to  determine  the  relevant  variables  in  a  problem, 
although  it  is  unclear  whether  they  are  learning  a  qeneral  or  t ask -spec i f i c 
skill. 

The  general i zabi I i ty  of  this  skill  to  unstructured  situations  is  also 
unclear.  We  do  suggest,  however,  that  techniques  like  the  Kelly  grid, 
factor  analysis,  and  group  problem  solving  will  help  in  generating  the 
variables  or  alternatives  for  a  particular  problem.  A  rule  of  thumb  that 
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seems  relevant  is  that  when  we  don't  know  how  to  train  people  in  a  partic¬ 
ular  skill,  we  should  give  them  practical  experience  in  the  skill,  in  hopes 
that  there  is  some  unart iculated  ability  that  will  be  acquired  through 

mere  practice. 

Defining  the  Criteria.  Goals,  or  Value  Dimensions.  When  the  decision 
is  not  well  structured,  that  is,  when  all  the  relevant  aspects  are  not 
explicitly  specified,  difficulties  arise.  Foremost  among  these  is  the 
neglect  of  one  or  more  crucial  factors  whose  relevance  only  becomes  apparent, 
sadly,  after  the  decision  has  been  made.  An  example  of  this  is  provided 
by  Birkin  and  Form  (1973),  who  examined  the  after-effects  of  the  "Zero 
Defects"  program.  This  program,  adopted  by  more  than  12,000  industrial 
firms,  attempted  to  attack  the  problem  of  defective  workmanship  by  mot  i  - 
vating  employees  to  do  the  job  right  the  first  time.  The  program  was 
based  upon  the  folloring  rationale:  "Because  of  the  complexity  of  today's 
products  and  because  of  the  drastic  consequences  of  product  failure,  manage¬ 
ment  should  use  all  means  possible  to  meet  customers'  specification.  Human 
error  on  the  job  is  not  inevitable  and  employees,  if  properly  motivated, 
could  maintain  a  desire  to  get  a  job  done  riqht  the  first  time."  Once 
the  program  was  implemented,  many  firms  discovered  they  could  not  live 
with  the  consequences  of  making  quality  a  primary  goal.  As  quality  rose, 
productivity  declined,  production  deadlines  were  missed  ahd  amounts  of 
spoiled  and  scrapped  goods  increased.  A  high  percentage  of  firms  dropped 
the  program. 

Although  there  are  a  number  of  techniques  for  specifying  criteria, 
goals,  and  value  attributes,  we  know  little  about  the  implications  of  these 
techniques  for  training.  For  example,  Jerry  Nadler  (1972)  has  developed 
a  widely  used  technique  called  function  expansion  which  is  supposed  to 
produce  a  set  of  mutually  exclusive  and  exhaustive  goals.  Despite  its 
popularity,  it  has  apparently  never  been  tested  or  compared  with  other 
processes.  A  suggested  technique  for  getting  people  to  reduce  the  number 
of  criteria  they  want  to  use  is  forcing  them  to  consider  carefully  the 
question  "What  will  you  really  want  to  know  ten  years  from  now?" 

Selecting  the  Structural  Mode  I .  Some  component  attitudes  and  skills 


are : 
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1.  Relieving  that  decomposition  results  in  better  judgments  and 
dec i s i ons . 

2.  Knowing  the  appropriate  level  of  decomposition  to  employ. 

3.  Charac te r i / i ng  .  by  flow  charts  or  some  other  technique, 
a  conceptual  picture  of  the  problem. 

I*.  Knowing  when  to  use  and  how  to  construct  simple  models. 

5.  Knowing  when  and  how  to  use  more  complex  mode  I s . 

6.  Knowing  how  to  draw  and  understand  decision  trees  and 
mat  r i ces . 

7.  Under s t and i ng  the  differences  between  normative  and  descriptive 
mode  I s . 

Training  for  these  skills  could  range  from  an  hour  and  a  half's 
“Idiot's  Guide"  approach  to  many  years  of  theoretic,  philosophic  and 
practical  study.  We  believe  that  these  skills  can  be  learned.  We  don't 
know  how  much  training  is  needed  before  a  decision  maker  can  handle  tasks 
like  lodeling  the  components  of  an  intricate  hierarchical  structure  or 
discovering  conditionality,  non i ndependence ,  and  redundancy  in  data.  We 
do  know  that  t he  visible  experts  in  the  field  have  Ph.D.'s 

Acquiring  I n* or  mat  ion.  The  typical  approach  to  information  collection 
is  gathering  a  mound  of  data  in  the  hope  that  you'll  have  what  you'll  need. 

In  Bayesian  information  purchase  experiments,  subjects  frequently  do  not 
buy  the  correct  amount  of  the  right  kind  of  information.  In  studies  comparing 
experienced  and  inexperienced  physicians,  the  latter  overcollect  irrelevant 
information.  Other  apparent  biases  are  over va I uat i on  o(  redundant  infor¬ 
mation,  preference  for  concrete  over  abstract  information  regardlrs 
d i agnos t i c i ty ,  failure  to  seek  negative  information,  and  underestimation 
of  the  diagnostic  value  of  information  whose  causal  implications  are  not 
obv i ous . 

Several  untested  techniques  were  proposed  for  training  proper  infor¬ 
mation  acquisition.  One  is  to  provide  the  cost  of  information  purchased. 

A  second  is  to  require  people  to  build  a  model  of  the  process  for  which 
they're  collecting  the  information  and  use  it  as  a  normative  standard  for 
data  collection.  Decision  makers  using  Bayesian  models,  for  example, 
would  be  required  to  determine  likelihood  ratios  as  well  as  information 
costs  and  then  perform  the  appropriate  calculat ions.  A  third  possible 


technique  is  tvavlnq  decision  makers  list  and  rank  the  questions  for  which 
they  would  have  liked  to  have  had  answers  before  they  made  their  last 
major  policy  decision.  Acquiring  Information  from  another  person  Is 
another  skill  that  may  be  trainable,  as  Is  recognizing  the  effects  of 
i ntercorrelat ions  between  envi ronmenta I  variables. 

Dealing  with  Uncertainty.  We  agreed  that  the  most  important  ideas 
and  attitudes  in  this,  the  most  heavily  researched  area,  are  (a)  that  the 
world  is  uncertain,  (b)  that  uncertainty  can  be  measured,  and  (c)  that 
dealing  loqically  with  this  uncertainty  will  make  a  difference  in  the 
quality  of  decision  making. 

One  topic  of  discussion  was  how  to  train  probability  assessors  to 
be  better  calibrated.  Suggestions  included  training  in: 

1.  Attendinq  to  relevant  information  and  iqnorinq  irrelevant 
informat  ion. 

2.  Recognizing  and  correcting  judgmental  biases. 

3.  Understand i ng  the  coherence  rules  of  probability. 

k  Understanding  the  equivalence  of  different  response  modes. 

5-  Using  group  situations  in  which  people  announce,  defend  and 
adjust  their  probabilities. 

6.  Using  scoring  rule  feedback. 

7.  Usinq  calibration  curves. 

Wo  also  agreed  that  it  was  important  that  probability  assessors  get  a 
qroat  deal  of  practice  assessing  probabilities  in  their  particular  field 
of  expertise.  The  examples  given  should  be  richly  detailed  and  as  close 
as  possible  to  the  kinds  of  data  that  the  decision  makers  would  meet  in 
the  field.  Furthermore,  the  examples  should  cover  the  probability  continuum 
of  low,  medium  and  high  likelihood  events.  Such  substantive  training  is 
especial ly  important  in  lieu  of  understanding  how  probability  traininq 
transfers  across  tasks. 

Weasur i ng  utility.  Although  the  skills  Involved  here  can  be  quite 
complex,  the  qroup  felt  that  little  effort  should  be  made  to  teach  either 
the  underlying  theory  or  fancy  applications.  Since,  for  the  most  part, 
additive  linear  utility  functions  work  quite  well,  people  should  be 
taught  enouqh  to  know  how  to  apply  the  linear  model  correctly  and  what 
special  c i rcumstances  signal  the  need  for  expert  assistance. 
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It  appears  that  one  way  that  decision  makers  avoid  difficulties  in 
situations  with  very  non-linear  utilities  is  with  the  "plausible  range- 
feasible  alternative"  approach.  That  is,  they  consider  as  feasible 
alternatives  only  those  that  lie  within  critical  threshold  values.  With¬ 
in  this  ranqe  of  plausible  values,  utility  functions  may  often  be  reasonably 
approximated  by  a  linear  utility  function.  Other  alternatives  are  not 
usually  even  analyzed.  A  danger  with  this  approach  is  that  decision  makers 
may  be  inappropriately  eliminating  highly  desirable  alternatives.  To 
reduce  this  danger,  we  should  train  decision  makers  to  identify  problem 
areas  where  this  is  likely  to  happen  and  teach  them  to  deal  with  these 
special  cases  by  using  non-linear  utility  functions  0  appropriate  heuristics. 

Combining  Single  Judgments  into  a  Composite  Judgment.  To  date,  neither 
judgment  theorists  nor  decision  analysts  have  answered  the  crucial  question 
in  this  area:  namely,  when  is  it  best  to  assess  components,  when  is  it 
best  to  make  global  judgments,  and  when  is  it  best  to  iterate  between 
these  two  procedures? 

Learning  from  Experience.  One  thing  that  must  be  taught  is  basic 
inductive  skills.  The  rules  of  scientific  research  advocated  by  philosophers 
of  science  were  suggested  as  one  source  of  insight  into  how  to  train 
i nduc  t i ve  skills. 

Another  direct  approach  to  teaching  such  ski' Is  is  to  alert  people 
to  avoid  the  known  biases  that  hinder  learning  from  past  experience, 
including  hindsight  bias  and  the  tendency  to  remember  one's  successes 
and  forqet  one's  failures.  Simply  informing  people  about  biases,  however, 
has  not  proven  successful.  What  is  needed  are  techniques  that  prevent 
perception  and  memory  distortions.  An  enthusiastically  received  suggestion 
made  at  this  meeting  is  to  have  people  set  up  record-keeping  systems  that 
carefully  document  the  considerations  preceding  their  own  decisions  and 
their  second  guesses  of  other  people's  decisions.  Such  a  record  cduld 
also  function  as  a  mechanized  data  file  from  which  actuarial  models  might 
eventually  be  developed.  One  potential  problem  with  this  system,  or  any 
other  system  involving  feedback,  is  that  feedback  can  be  inaccurate, 
misleading  or  deceptive.  As  mentioned  above,  good  decisions  and  good 
outcomes  are  not  invariably  associated  with  one  another.  Thus,  training 
in  setting  up  feedback  systems  would  have  to  include  rules  for  recognizing 
the  limitations  of  different  types  of  feedback.  One  example  of  elaborate 
record  keeping  is  football  coaching  records  where  every  play  is  analyzed 
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and  its  consequences  recorded.  Systematic  study  of  the  effectiveness  of 
record  keeping  is  needed. 

Existing  research  suggests  that  mere  instruction  is  insufficient  for 
the  learning  of  cognitive  strategies.  One  also  needs  lots  of  practice 
in  varied  contexts  over  time.  Each  cognitive  task,  may  have  an  optimum 
learning  program  including  lectures  and  book-learning,  highly  structured 
laboratory  exercises  with  accompanying  feedback,  and  guided,  but  less 
unstructured,  clinical  experience.  The  optimum  mix  of  these  components 
is  an  open  question. 

Knowing  When  and  How  to  Properly  Use  Heuristics,  Shortcuts,  Decision 
Aids  and  Normative  Experts.  As  research  evidence  is  scanty  here,  practice 
appears  to  be  the  only  tested  means  of  learning  these  skills.  Some  inter¬ 
esting  rules  of  thumb:  it  was  reported  that  Ron  Howard  has  recommended 
that  an  organization  should  devote  I?  of  the  resources  of  a  project  to 
decision  analysis.  (if  applied  to  the  U.S.  Gross  National  Product,  this 
rule  would  provide  over  $10  billion  in  income  for  decision  analysts.) 

Some  educators  believe  that  the  probability  that  a  decision  aid  or  model 
will  t»e  used  is  directly  proportional  to  its  ease  of  use.  There  arc 
•’'any  c  ha  rac  ter  i  zab  1  e  situations  where  decision  making  should  be  left  to 
an  algebraic  model,  or  discrimination  network-type  model,  or  simulation 
Such  situations  usually  involve  repeatable  tasks  with  a  narrow  focus. 

Some  of  the  issues  involving  when  to  use  each  of  the  various  types 
of  decision  aids  are  discussed  by  Slovic  (1977).  Automated  decision- 
analytic  procedures  create  another  whole  category  of  training  issues, 
particularly  getting  people  to  accept  the  procedure  and  use  its  results. 
Acceptance  may  be  particularly  problematic  when  the  advice  of  the  automated 
procedure  is  in  some  way  counterintuitive. 

Conpunicat ion,  Monitoring,  and  Implementation.  It  is  one  thing  to 
train  decision  makers  to  make  better  decisions  when  they  have  total  control 
over  their  analyses.  The  situation  changes  when  the  decision  maker  is 
dependent  on  others  to  gather  information,  generate  alternatives  or 
accomplish  some  other  aspect  of  the  decision  analysis.  Crucial  skills, 
then,  are  communication,  monitoring,  and  implementation.  Organi zat ional 
and  social  psychologists  and  managerial  specialists  are  the  appropriate 
sources  of  this  knowledge  (e.g.,  Shultz  and  Slevln,  1975;  ludke,  1976; 
Dalton.  1969;  and  Utterback,  I97M- 


III.  Important  Research  Questions 

The  most  important  questions  for  future  research  that  evolved  during 
the  two  days  of  discussion  were: 

1.  How  do  you  define  a  "good"  decision  when  you  don’t  have  an 
objective  criterion? 

2.  How  do  you  measure  the  effectiveness  of  a  decision  maker?  How 

do  you  know  when  you  have  improved  the  effectiveness  or  efficiency 
of  a  decision  maker  through  training? 

3.  Which  combinations  of  training  general  principles  and  specific 
techniques  are  best  to  bring  decision  makers  to  different 
levels  of  competency? 

k.  Can  we  develop  a  diagnostic  (interviewing)  technique  for 
surveying  a  particular  decision-making  context  to  find  out 
where  training  should  be  concentrated? 

5.  How  do  the  various  different  training  programs  used  today  differ? 
What  are  the  virtues,  difficulties,  and  limitations  of  each? 

What  is  the  effectiveness,  retention,  and  transferability  of 
each  of  the  skills  I  earned  in  each  of  these  programs? 

6.  When  does  the  use  of  the  decision  analytic  methodology  improve 
clinical  j  udgment  ? 

7.  How  much  apprenticeship  is  required  to  train  a  skilled  decision 
analyst,  in  terms  of  time,  number  of  supervised  cases,  and 
varieties  of  case  experience? 

3.  Would  it  be  worthwhile  (in  a  cost-benefit  sense)  to  implement 
the  problem  -or i ented  medical  record  procedure  in  other  areas, 
such  as  the  military? 

9.  How  do  you  traing  people  to  detect  the  dimensions  of  any  unstruc¬ 
tured  situation?  Is  training  for  comprehensiveness  detrimental 
to  training  for  focussing  on  the  important  variables? 

10.  How  do  you  train  people  to  evaluate  the  finality  or  the  complete¬ 
ness  of  their  search  for  the  dimensions  or  the  variables  of  a 
problem? 

11.  When  should  decision  aids,  such  as  interactive  computer  programs, 
substitute  for  the  more  intensive  personal  training  of  decision 
makers?  Is  such  training  cost  effective?  When  is  training  the 
decision  maker  in  the  theory  of  decision  making  cost  effective? 
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12.  How  do  you  train  decision  makers  to  make  proper  tradeoffs  between 
model  sophistication  and  simplification,  keeping  in  mind  the 
widely-held  belief  that  model  use  is  directly  proportional  to 
ease  of  use? 

13.  How  should  we  train  decision  makers  to  acquire  information? 

IA.  In  probability  assessment,  what  is  the  appropriate  role  of  the 

decision  maker  and  the  decision  analyst?  Similarly,  how  different 
are  the  estimates  produced  by  decision  makers  and  the  analysts? 

To  what  extent  do  different  analysts  differ? 

15.  When  is  it  best  to  assess  utility  components,  and  when  best  to 
obtain  global  Judgments? 

16.  Psychologists  have  uncovered  many  biases  which  people  have  that 
can  hinder  their  ability  to  make  decisions.  What  techniques  can 
be  used  to  overcome  these  biases? 

17.  What  are  the  essential  features  of  a  record-keeping  system  that 
would  document  a  decision  analysis  and  its  eventual  outcome? 

18.  What  are  guidelines  for  the  appropriate  use  of  heuristics, 
short  cuts,  decision  aids  and  normative  experts? 

IV.  Recommendations 

Given  the  current  state  of  our  knowledge,  three  aspects  of  the  training 
problem  demand  immediate  attention:  Traininq  specific  skills,  evaluating 
the  quality  of  decisions,  and  implementing  the  knowledge  obtained  through 
decision  research.  Although  these  three  areas  arc  substantially  inter¬ 
dependent,  the  research  strategies  each  entails  are  sufficiently  different 
to  merit  distinguishing  between  them. 

Training  Specific  Skills 

Tasks  included  here  are:  identifying  judgmental  biases,  character¬ 
izing  known  biases  in  terms  of  underlying  cognitive  processes,  determining 
which  biases  are  amenable  to  training  and  which  can  only  be  compensated 
for  mechan i ca 1 1 y ,  developing  dehiasing  or  b ias-avoi d i ng  procedures  where 
applicable,  and  developing  guidelines  for  characterizing  decision  situations. 

The  research  skills  required  here  are  those  of  experimental  psychology. 
Althouqh  the  problems  are  of  an  applied  nature,  it  appears  to  us  that  they 
will  be  most  parsimoniously  attacked  by  looking  for  conmon  elements  in  the 
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deci s ion -making  strategies  used  in  different  tasks.  These  are,  we  believe, 
basic  cognitive  processes.  When  a  debiasing  procedure  proves  successful, 
it  means  that  we  have  identified  the  bias,  understood  its  workings  and 
communicated  that  understanding  to  the  decision  maker  in  a  useful  form. 

When  it  proves  unsuccessful,  it  is  hard  to  know  where  we  went  wrong.  The 
failure  of  the  relatively  few  attempts  which  have  been  made  to  reduce 
biases  by  simply  telling  people  about  them  suggest  that  the  problems  may 
be  nere  complex  than  we  have  realized.  The  sort  of  cognitive  engineering 
we  are  concerned  with  will  require  considerable  interaction  with  basic 
cognitive  research,  digging  deeper  in  the  mind  when  more  straightforward 
debiasing  procedures  fail. 

Such  training  studies  as  are  conducted  should  be  tightly  controlled 
to  maximize  the  chances  of  our  being  able  to  tell  what  works  and  what 
doesn't  work  in  this  di f f icul t - to- resea rch  area.  For  similar  reasons, 
we  advocate  concentrating  on  the  training  of  specific  skills  rather  than 
the  development  of  full  programs.  The  latter  are  typically  more  expensive 
and  context -speci f i c ,  and  offer  less  opportunity  for  clearly  assessing 
what  are  the  effective  and  ineffective  elements. 

Eva  I uat i ng  the  Quality  of  Decisions 

A  recurrent  theme  of  the  conference  was  that  we  do  not  know  how  to 
evaluate  most  important  decisions.  Without  such  knowledge,  there  is  no 
way  of  assessing  the  value  of  the  various  training  proqrams  now  offered 
or  the  validity  of  the  claims  made  for  them.  At  the  extreme,  we  do  not 
even  know  if  any  simple  decision-making  procedures,  however  flimsy  its 
axiomatic  basis,  is  not  as  good  as  the  most  sophisticated. 

Developing  evaluative  criteria  is  a  difficult,  long-term  project.  It 
should  be  accompanied  by  an  attempt  to  survey  and  compare  existing  programs 
using  the  best  tools  currently  available.  Such  a  survey  should  include 
conmerc i a  I  I y-marketed  nonaxiomatic  approaches  like  Jerry  Nadler's  and 
Dale  Carnegie's.  We  should  not  exclude  the  possibility  that  they  know 
something  that  we  don't. 

Another  aspect  of  the  evaluation  problem  is  to  assess  the  importance 
of  the  judgmental  biases  that  have  been  identified  by  various  investigators. 


19 


that  is,  to  determine  how  much  of  a  difference  they  make  in  the  optimality 
of  decisions.  What  is  called  for  here  is  some  general  work  on  the  sensi¬ 
tivity  of  decisions  to  bias.  It  should  consider  the  apparently  untouched 
question  of  how  bias  and  error  in  the  various  component  judgments  used 
in  a  comple*  analysis  combine  to  affect  the  final  outcome.  Some  analysis 
of  the  ecological  validity  of  judgmental  heuristics  would  also  be  useful 
in  order  to  establish  the  frequency  and  extent  of  the  biases  they  induce. 

A  final  aspect  of  the  evaluation  problem  is  developing  ways  to  help 
people  best  assess  the  quality  of  their  own  decisions  (and  those  of  their 
colleagues),  and  learn  from  their  own  experience.  Hindsight  research 
(Fischhoff,  1975)  suqqests  some  ways  in  which  this  miqht  be  done,  as  do 
the  decision  record-keeping  systems  described  by  Shulman  and  Shanteau. 
Implement i  Present  Knowledge 

There  is,  already,  much  knowledge  that  can  be  used  to  train  decision 
makers  and  improve  the  quality  of  their  performance.  For  repeatable 
tasks,  the  heretofore  covert  decision  processes  of  the  expert  can  be 
modeled  and  made  explicit  in  a  way  t ha t  should  be  quite  useful  to 
trainees.  In  some  cases,  these  models  will  take  the  form  of  algebraic 
equations.  In  others,  more  complex  models,  such  as  sequentially  branching 
computer  programs,  will  be  necessary.  The  potential  of  judgment  modeling 
for  facilitating  military  and  defense  decisions  is  unlimited.  Demonstration 
projects  should  begin  immediately.  Relevant  background  information  is 
provided  by  Hammond  ct  al.  (1975)  and  Slovic  and  Lichtenstein  (1971). 

The  majority  of  important  decisions  will  be  made  in  unique  situations 
that  permit  little  time  for  del iberat ion.  The  standard  method  of  decisions 
in  these  situations  will  continue  to  be  intuitive.  Given  the  pitfalls 
to  which  unaided  intuitions  are  susceptible,  we  have  little  reason  to  be 
comfortable  with  this  prospect.  Since  we  cannot  avoid  heavy  reliance 
on  intuition,  we  should  at  least  educate  decision  makers  to  the  pitfalls 
that  await  the  unwary.  We  recommend  that  consideration  be  qiven  to  the 
development  of  a  curriculum  designed  to  educate  the  intuitions  of  decision 
makers  in  the  subtleties  of  dealing  with  a  complex  and  uncertain  environment. 
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